Fast k-Nearest Neighbour Search via Prioritized DCI

Most exact methods for k-nearest neighbour search suffer from the curse of dimensionality; that is, their query times exhibit exponential dependence on either the ambient or the intrinsic dimensionality. Dynamic Continuous Indexing (DCI) (Li & Malik, 2016) offers a promising way of circumventing the curse by avoiding space partitioning and achieves a query time that grows sublinearly in the intrinsic dimensionality. In this paper, we develop a variant of DCI, which we call Prioritized DCI, and show a further improvement in the dependence on the intrinsic dimensionality compared to standard DCI, thereby improving the performance of DCI on datasets with high intrinsic dimensionality. We also demonstrate empirically that Prioritized DCI compares favourably to standard DCI and Locality-Sensitive Hashing (LSH) both in terms of running time and space consumption at all levels of approximation quality. In particular, relative to LSH, Prioritized DCI reduces the number of distance evaluations by a factor of 5 to 30 and the space consumption by a factor of 47 to 55.